Algorithms for Nash Equilibria in General-Sum Stochastic Games

Over the past few decades the quest for algorithms to compute Nash equilibria in general-sum stochastic games has intensified and several important algorithms (cf. [9], [12], [16], [7]) have been proposed. However, they suffer from either lack of generality or are intractable for even medium sized problems or both. In this paper, we first formulate a non-linear optimization problem for stochastic games and then break it down into simpler subproblems that ensure there is no Bellman error for a given state and agent. Next, we derive a set of novel necessary and sufficient conditions for solution points of these sub-problems to be Nash equilibria of the underlying game. Using these conditions, we develop two novel algorithms OFF-SGSP and ON-SGSP,respectively. OFF-SGSP is an off-line centralized algorithm which assumes complete information of the game. On the other hand, ON-SGSP is an online decentralized algorithm that works with simulated transitions of the stochastic game. Both algorithms are guaranteed to converge to Nash equilibrium strategies for general-sum (discounted) stochastic games.